Program Evolution for General Intelligence

A program evolution component is proposed for integrative artificial general intelligence. The system’s deployment is intended to be comparable, on Marr’s level of computational theory, to evolutionary mechanisms in human thought. The challenges of program evolution are described, along with the requirements for a program evolution system to be competent solving hard problems quickly, accurately, and reliably. Meta-optimizing semantic evolutionary search (MOSES) is proposed to fulfill these requirements. 1. Background and Motivation “At every step the design logic of brains is a Darwinian logic: overproduction, variation, competition, selection ... it should not come as a surprise that this same logic is also the basis for the normal millisecond-by-millisecond information processing that continues to adapt neural software to the world.” Terrence Deacon [1] In David Marr’s seminal decomposition, any information-processing system may be understood at three nearly independent levels: (1) computational theory, a description of the problems the system attempts to solve; (2) representations and algorithms; and (3) implementation, the physical instantiation of the system’s representations and algorithms [2]. What might the subsystems of human cognition look like on the level of computational theory?1 I propose that evolutionary learning be considered as one of these subsystems. Major subfields of both cognitive science and AI are concerned with evolutionary learning processes. One motivator for this concern is an attempt in both fields to augment purely local techniques such as Hebbian (associative) learning with more global methods, which try to make large leaps to find answers far removed from existing knowledge. This is a form of evolutionary learning [4], which Edelman [5] has presented as “Neural Darwinism”, and Calvin and Bickerton [6, 7] as the notion of mind as a “Darwin Machine”. It is known that the immune system adapts via a form of evolutionary learning, and Edelman [5] has proposed that the brain does so as well, evolving new “neuronal maps”, patterns of neural connection and activity spanning numerous neuronal clusters that are highly “fit” in the sense of contributing usefully to system goals. Edelman and 1 Parts of this section are adapted from [3], which elaborates on the utility of emulating human cognition on the level of computational theory. his colleagues have run computer simulations showing that Hebbian-like neuronal dynamics, if properly tuned, can give rise to evolution-like dynamics on the neuronal map level (“neuronal group selection”). Recently Deacon [1] has articulated ways in which, during neurodevelopment, difference computations compete with each other (e.g., to determine which brain regions are responsible for motor control). More generally, he posits a kind of continuous flux as control shifts between competing brain regions, again, based on high-level “cognitive demand” [1, p. 457]. Similarly, Calvin and Bickerton [6, 7] have given plausible neural mechanisms (“Darwin Machines”) for synthesizing short “programs”. These programs are for tasks such as rock throwing and sentence generation, which are represented as coherent firing patterns in the cerebral cortex. A population of such patterns, competing for neurocomputational territory, replicates with variations, under selection pressure to conform to background knowledge and constraints. In summary, a system is needed that can recombine existing solutions in a nonlocal synthetic fashion, learning nested and sequential structures, and incorporate background knowledge (e.g. previously learned routines). I propose a particular kind of program evolution to satisfy these goals. 1.1. Evolutionary Learning There is a long history in AI of applying evolution-derived methods to practical problem-solving; the original genetic algorithm [4], initially a theoretical model, has been adapted successfully to a wide variety of applications [8]. The methodology, similar to the Darwin Machines mentioned above, is applied as follows: (1) generate a random population of solutions to a problem; (2) evaluate the solutions in the population using a predefined scoring function; (3) select solutions from the population proportionate to their scores, and recombine/mutate them to generate a new population; (4) go to step 2. Holland's paradigm has been adapted from the case of fixed-length strings to the evolution of variable-sized and shaped trees (typically Lisp symbolic expressions), which in principle can represent arbitrary computer programs [9, 10]. Recently, replacements-for/extensions-of the genetic algorithm have been developed (for fixed-length strings) which may be described as estimation-ofdistribution algorithms (see [11] for an overview). These methods, which outperform genetic algorithms and related techniques across a range of problems, maintain centralized probabilistic models of the population learned with sophisticated datamining techniques. One of the most powerful of these methods is the Bayesian optimization algorithm (BOA) [12]. The basic steps of the BOA are: (1) generate a random population of solutions to a problem; (2) evaluate the solutions in the population using a predefined scoring function; (3) from the promising solutions in the population, learn a generative model; (4) create new solutions using the model, and merge them into the existing population; (4) go to step 2. The neurological implausibility of this sort of algorithm is readily apparent – yet recall that we are attempting to emulate human cognition on the level of computational theory, not implementation, or even representations and algorithms. Figure 1: The structure of OneMax, a paradigmatic separable optimization problem. Figure 2: The structure of hierarchical if-and-only-if [16], a paradigmatic nearly decomposable optimization problem. Figure 3: The structure of an intractable optimization problem, such as a uniform random scoring function, where changing the assignment of any variable results in a chaotic change in the overall score. Fundamentally, the BOA and its ilk (the competent adaptive optimization algorithms) differ from classic selectorecombinative search by attempting to dynamically learn a problem decomposition, in terms of the variables that have been pre-specified. The BOA represents this decomposition as a Bayesian network (directed acyclic graph with the variables as nodes, and an edge from x to y indicating that y is probabilistically dependent on x). An extension, the hierarchical Bayesian optimization algorithm (hBOA) [12], uses a Bayesian network with local structure [13] to more accurately represent hierarchical dependency relationships. The BOA and hBOA are scalable and robust to noise across the range of nearly decomposable functions [12, 13]. They are also effective, empirically, on real-world problems with unknown decompositions, which may or may not be effectively representable by the algorithms; robust, high-quality results have been obtained for Ising spin glasses and MaxSAT [14], as well as a real-world telecommunication problem [15]. 1.2.Representation-Building “A representation is a formal system making explicit certain entities or types of information, together with a specification of how the system does this.” David Marr [2] In an ideally encoded optimization problem, all prespecified variables would exhibit complete separability, and could be optimized independently (Figure 1). Problems with hierarchical dependency structure (Figure 2) cannot be encoded this way, but are still tractable by dynamically learning the problem decomposition (as the BOA and hBOA do). For complex problems with interacting subcomponents, finding an accurate problem decomposition is often tantamount to finding a solution. In an idealized run of a competent optimization algorithm, the problem decomposition evolves along with the set of solutions being considered, with parallel convergence to the correct decomposition and the global solution optima. However, this is certainly contingent on the existence of some compact2 and reasonably correct decomposition in the space (of decompositions, not solutions) being searched. Difficulty arises when no such decomposition exists (Figure 3), or when a more effective decomposition exists that cannot be formulated as a probabilistic model over representational parameters. Accordingly, one may extend current approaches via either: (1) a more general modeling language for expressing problem decompositions; or (2) additional mechanisms that modify the representations on which modeling operates (introducing additional inductive bias). I focus here on the latter – the former would appear to require qualitatively more computational capacity than will be available in the near future. If one ignores this constraint, such a “universal” approach to general problem-solving is indeed possible [17, 18]. I refer to these additional mechanisms as representation-building because they serve the same purpose as the pre-representational mechanisms employed (typically by humans) in setting up an optimization problem – to present an optimization algorithm with the salient parameters needed to build effective problem decompositions and vary solutions along meaningful dimensions. A secondary source of human effort in encoding problems to be solved is crafting an effective scoring function. The primary focus of this paper is issues surrounding the representation of solutions, rather than how solutions are scored, which may be more fruitfully addressed, I believe, in the context of integrative systems (cf. [19]). 1.3.Program Learning An optimization problem may be defined as follows: a solution space S is specified, together with some scoring function on solutions, where “solving the problem” corresponds to discovering a solution in S with a sufficiently high score. Let's define program learning as follows: given a program space P, a behavior space B, an execution function exec : P → B, and a scoring function on behaviors, “solving the problem” corresponds to discovering a program p in P whose corresponding behavior, exec(p), has a sufficiently high score. This extended formalism can of course be entirely vacuous – the behavior space could be identical to the program space, and the execution function simply identity, allowing any optimization problem to be cast as a problem of program learning. The utility of this specification arises when we make interesting assumptions regarding the program and behavior spaces, and the execution and scoring functions (the additional inductive bias mentioned above): 2 The decomposition must be compact because in practice only a fairly small sampling of solutions may be evaluated (relative to the size of the total space) at a time, and the search mechanism for exploring decomposition-space is greedy and local. This is in also accordance with the general notion of learning corresponding to compression [17]. • Open-endedness – P has a natural “program size” measure – programs may be enumerated from smallest to largest, and there is no obvious problemindependent upper bound on program size. • Over-representation – exec often maps many programs to the same behavior. • Compositional hierarchy – programs themselves have an intrinsic hierarchical organization, and may contain subprograms which are themselves members of P or some related program space. This provides a natural family of distance measures on programs, in terms of the the number and type of compositions / decompositions needed to transform one program into another (i.e., edit distance). • Chaotic Execution – very similar programs (as conceptualized in the previous item) may have very different behaviors. Precise mathematical definitions could be given for all of these properties but would provide little insight – it is more instructive to simply note their ubiquity in symbolic representations; human programming languages (LISP, C, etc.), Boolean and real-valued formulae, pattern-matching systems, automata, and many more. The crux of this line of thought is that the combination of these four factors conspires to scramble scoring functions – even if the mapping from behaviors to scores is separable or nearly decomposable, the complex3 program space and chaotic execution function will often quickly lead to intractability as problem size grows. These properties are not superficial inconveniences that can be circumvented by some particularly clever encoding. On the contrary, they are the essential characteristics that give programs the power to compress knowledge and generalize correctly, in contrast to flat, inert representations such as lookup tables (see Baum [20] for a full treatment of this line of argument). The consequences of this particular kind of complexity, together with the fact that most program spaces of interest are combinatorially very large, might lead one to believe that competent program evolution is impossible. Not so: program learning tasks of interest have a compact structure4 – they are not “needle in haystack” problems or uncorrelated fitness landscapes, although they can certainly be encoded as such. The most one can definitively state is that algorithm foo, methodology bar, or representation baz is unsuitable for expressing and exploiting the regularities that occur across interesting program spaces. Some of these regularities are as follows: • Simplicity prior – our prior assigns greater probability mass to smaller